Machine learning apparatus, imaging time estimation apparatus, machine learning program, and generation method of photographic data

ABSTRACT

A machine learning apparatus includes a learning target obtaining unit that obtains a plurality of photographic data items of scanned photographs and a learning unit that learns imaging times of the photographs in which imaging dates are not imprinted based on the photographic data items.

The present application is based on, and claims priority from JPApplication Serial Number 2018-091951, filed May 11, 2018, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a machine learning apparatus, animaging time estimation apparatus, a machine learning program, and ageneration method of photographic data.

2. Related Art

In the related art, there is a need for the rearrangement of a pluralityof scanned photographs in order of date. For example, JP-A-2004-348620discloses a technology in which a negative film and a photographic printare scanned, the same image is searched for from the scanning result,and an imprinted date extracted from the negative film is specified asan imaging date of the print image.

It is not possible to specify an imaging time of a photograph in whichan imaging date is not imprinted.

SUMMARY

A machine learning apparatus includes an obtaining unit that obtainsphotographic data items of scanned photographs, and a learning unit thatlearns imaging times of the photographs in which imaging dates are notimprinted based on the photographic data items. According to theaforementioned configuration, even though the imaging date is notimprinted in the photograph, it is possible to estimate the imaging timewhen the photograph is captured.

The learning unit may perform the learning based on training data itemsobtained by associating the photographic data items with the imagingtimes of the photographs. According to this configuration, it ispossible to estimate the imaging time of the photograph even though therelationship between the feature of the photograph and the imaging timeis not clearly defined by human judgment.

The training data items may include data items obtained by associatingthe photographic data items obtained by scanning the photographs inwhich imaging dates are imprinted with imaging times indicating theimprinted imaging dates. That is, when the imaging date is imprinted andthus, the photographic data is generated by scanning the photograph ofwhich the imaging date is clearly specified, it is possible to generatethe training data having a high possibility that an accurate output isto be obtained by associating the imaging date with the photographicdata.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a machine learning apparatus and an imageprocessing apparatus.

FIG. 2 is a flowchart illustrating machine learning processing.

FIG. 3 is a diagram illustrating a structure of a neural network.

FIG. 4 is a diagram illustrating an example of a model.

FIG. 5 is a flowchart illustrating image processing.

FIG. 6 is a diagram illustrating an example of an interface screen.

FIG. 7 is a flowchart illustrating image processing.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, an embodiment of the present disclosure will be describedaccording to the following order.

(1) Configuration of Machine Learning Apparatus and Image ProcessingApparatus:

(2) Machine Learning Processing:

(3) Image Processing:

(4) Other Embodiments:

(1) Configuration of Machine Learning Apparatus and Image ProcessingApparatus

FIG. 1 is a block diagram illustrating a machine learning apparatus andan image processing apparatus according to an embodiment of the presentdisclosure. The machine learning apparatus and the image processingapparatus according to the present embodiment is realized by a computer1 to which a scanner 2 is connected. The computer 1 includes a processor10, a hard disk drive (HDD) 20, a device interface 30, and a displayinterface 40.

The device interface 30 is an interface configured to connect to adevice that performs communication according to a predeterminedcommunication protocol (for example, a Universal Serial Bus (USB)standard). In the present embodiment, the scanner 2, a mouse 3 a, and akeyboard 3 b are connected to the device interface 30. The scanner 2includes a light source that illuminates light on a document, a colorimage sensor that receives light from the document and uses the receivedlight as read data, a device component for moving various movable units,and the like.

In the present embodiment, the scanner 2 can read printed photographs,and can output photographic data items. The computer 1 obtains thephotographic data items output by the scanner 2 through the deviceinterface 30, and stores photographic data items 20 a in the HDD 20.There may be various aspects as the method of reading the document inthe scanner 2. A flatbed scanner may be used, a method of reading atransport document using an ADF may be used, or both the methods may beused.

The mouse 3 a includes an operation unit and a button that is held andmoved by a user, and outputs information indicating a movement amount ofthe operation unit and information indicating a result of the operationfor the button. The computer 1 can obtain information output by themouse 3 a through the device interface 30, and can receive the operationof the user based on the information. The keyboard 3 b includes aplurality of keys, and outputs information indicating an operation forthe key. The computer 1 can obtain the information output by thekeyboard 3 b through the device interface 30, and can receive theoperation of the user based on the information.

The display interface 40 is an interface to which a display 4 isconnected. The computer 1 is configured to display various images on thedisplay 4 by outputting a control signal to the display 4 through thedisplay interface 40.

The processor 10 includes a CPU, a RAM, and the like, and can executevarious programs stored in the HDD 20. In the present embodiment, amachine learning program 11 and an image processing program 12 areincluded in these programs. The machine learning program 11 is a programcausing the processor 10 to perform a function of learning an imagingtime of a photograph in which an imaging date is not imprinted based onthe photographic data items of the scanned photographs. When the machinelearning program 11 is executed, the processor 10 functions as alearning target obtaining unit 11 a and a learning unit 11 b.

The image processing program 12 is a program causing the processor 10 toperform a function of calculating degrees of fading of the photographsbased on the photographic data items obtained by scanning thephotographs, and organizing the photographic data items according todegrees of oldness of the photographs by deeming the photograph of whichthe degree of fading is larger to be an old photograph. When the imageprocessing program 12 is executed, the processor 10 functions as anobtaining unit 12 a, a calculating unit 12 b, a reading unit 12 c, andan organizing unit 12 d.

(2) Machine Learning Processing

Next, machine learning processing performed by the processor 10 will bedescribed. FIG. 2 is a flowchart illustrating the machine learningprocessing. The machine learning processing is performed in advancebefore the imaging time of the photograph is estimated. When the machinelearning processing is started, the processor 10 obtains a trainingmodel by the function of the learning unit 11 b (step S100). In thisexample, the model is information indicating an expression for derivinga correspondence between data of an estimation target and data of anestimation result. In the present embodiment, a model in which thephotographic data items or the data items (the degrees of fading)derived from the photographic data items are associated with imagingtime will be described as an example.

The model can be defined in various manners as long as input data itemsare converted into output data items. In the present embodiment, aneural network is included in at least a part of the model. FIG. 3 is aschematic diagram illustrating a structure of a neural network that mayinclude the model used in the present embodiment. In FIG. 3, nodesmodeling neurons are represented by circles, and connections between thenodes are represented by straight lines as solid lines (in this regard,a part of the nodes and the connections are represented for the sake ofsimplicity of illustration). In FIG. 3, the nodes belonging to the samelayer are represented so as to be vertically lined up, a layer at a leftend is an input layer Li, and a layer at a right end is an output layerLo.

In this example, in the relationship between two layers, a layer closerto the input layer Li is referred to as a higher layer, and a layercloser to the output layer Lo is referred to as a lower layer. That is,a structure in which outputs of a layer which is higher than a certainlayer by one layer are inputs for the certain layer and values obtainedby respectively multiplying the inputs by weights and respectivelyadding biases to the multiplied resultant values are output through anactivation function is illustrated. For example, when the number ofnodes of a layer L_(P) illustrated in FIG. 3 is P, input data items foreach node of the layer L_(P) are x₁, , , , , x_(M), biases are b₁, , , ,, b_(P), weights for a k-th node of the layer L_(P) are w_(k1), , , ,w_(kM), and the activation function is h(u), intermediate outputs u₁, ,, , , u_(P) of each node of the layer L_(P) are expressed by thefollowing Expression 1.

$\begin{matrix}{\left( {u_{1}\mspace{14mu}\ldots\mspace{14mu} u_{k}\mspace{14mu}\ldots\mspace{14mu} u_{P}} \right) = {{\left( {x_{1}\mspace{14mu}\ldots\mspace{14mu} x_{k}\mspace{14mu}\ldots\mspace{14mu} x_{M}} \right)\begin{pmatrix}w_{11} & \ldots & w_{k\; 1} & \ldots & w_{P\; 1} \\\; & \; & \vdots & \; & \; \\w_{1\; k} & \ldots & w_{kk} & \ldots & w_{Pk} \\\; & \; & \vdots & \; & \; \\w_{1\; M} & \ldots & w_{kM} & \ldots & w_{PM}\end{pmatrix}} + \left( {b_{1}\mspace{14mu}\ldots\mspace{14mu} b_{k}\mspace{14mu}\ldots\mspace{14mu} b_{P}} \right)}} & (1)\end{matrix}$

Accordingly, when Expression 1 is substituted into the activationfunction h(uk), outputs y₁, , , , , y_(P) of the nodes of the layerL_(P) are obtained.

The model is constructed by providing a plurality of nodes generalizedin this manner. Accordingly, in the present embodiment, the model isconstituted by at least data items indicating the weights, the biases,and the activation functions in each layer. Various functions, forexample, a sigmoid function, hyperbolic function (tan h), rectifiedlinear unit (ReLU), and other functions can be used as the activationfunction. A nonlinear function may be used. Of course, other conditionsnecessary for machine learning, for example, parameters such as kinds orlearning rates of optimization algorithms may be included. There may bevarious aspects as the connection relationship of the number of layersor the number of nodes, and the correlation.

In the present embodiment, a model in which the output data itemsindicating the imaging times of the photographs are output from theoutput layer Lo by using the photographic data items or the data itemsderived from the photographic data items as input data items for theinput layer Li is constructed. For example, in the example illustratedin FIG. 3, an example in which the input data items of each node of theinput layer Li are gradation values of each pixel of the photographicdata items (values by standardizing the gradation values thereof) andthe output data items from the output layer Lo correspond to the rangesof the imaging time can be assumed. An example of the configuration inwhich the imaging time is output is a configuration in which the timeranges such as 1900s, 1910s, and 1920s are assigned to the nodes and therange assigned to the node that outputs the maximum value is deemed tobe the imaging time of the input photograph.

Of course, a part of the model illustrated in FIG. 3 may be expressed bya known function, the remaining part thereof may be expressed by anunknown function, and the unknown function may be a learning target. Inany case, when the model in which the photographic data items are inputand the imaging time is ultimately output is constructed, the model forestimating the imaging times from the photographic data items can belearned. Although it has been described in the present embodiment thatthe model is constructed based on such an idea, this idea is based onthe fact that the color of the photographic print fades over time.

That is, in the present embodiment, the degree of oldness of thephotograph is deemed to correspond to a time interval between theprinted date and the current date. In the present embodiment, as thetime interval between the printed date and the current date is longer,the degree of fading is deemed to be large.

In the present embodiment, the printed date and the imaging date arerarely deemed to be greatly different (for example, for every severalyears). By deeming in this manner, the degree of fading of thephotograph and the degree of oldness of the imaging time can be deemedto correspond to each other. When the photographic data is input and thedegree of fading is calculated, the degree of oldness of the photograph,that is, the imaging time can be estimated through this calculation.

According to such an idea, when the neural network illustrated in FIG. 3is learned such that the photographic data items are input to the inputlayer Li and the imaging times is output from the output layer Lo, it isconsidered that the values indicating the degrees of fading are outputfrom a layer which is higher than the output layer by one or morelayers. In order to enable the estimation of the imaging times throughthe degrees of fading, the layer that outputs the degrees of fading maybe automatically learned as a hidden layer in the machine learning, orthe degrees of fading may be more explicitly introduced into the model.

For example, a neural network in which a function that calculates thedegrees of fading by inputting the photographic data items is preparedin advance and the degrees of fading are output may be constructed. Forexample, such a model can be constructed by expressing layers betweenthe input layer Li of the model illustrated in FIG. 3 and a layer L_(F)that outputs the degrees of fading by a known function f. That is, amodel in which the function f that calculates the degrees of fading isdefined from the gradation values (Ri, Gi, Bi) of the photographic dataitems in advance and outputs Fi of the function are used as outputs forthe layer L_(F) may be constructed.

FIG. 4 is a schematic diagram illustrating a model in which the valuesFi indicating the degrees of fading are calculated by using the functionf. That is, when gradation values of color channels of each pixel ofcertain photographic data i are (Ri, Gi, Bi) and these gradation valuesare input to the function f, output values F_(i1) to F_(iN) indicatingdegrees of fading are obtained. When the output values F_(i1) to F_(iN)indicating the degrees of fading are input to the layer L_(F) and amodel between the Layer L_(F) and the output layer Lo is optimized bythe machine learning, a model in which information items (T_(i1) toT_(i0)) indicating the imaging times of the photographs are output fromthe degrees of fading of the photograph can be learned.

The function f that calculates the degree of fading may be defined byvarious methods. The degree of fading may be evaluated by variousmethod. For example, when a contrast width of a color (yellow or thelike) which tends to deteriorate on the printed photograph is analyzed,it is possible to determine that as the contrast of this color is lower,the degrees of fading are large. The color of interest may be at leastone of color components of the photographic data items, or may be acolor (yellow or the like) calculated from two or more color components.Of course, a target of interest may be a target other than hue, and thedegrees of fading may be determined based on contrast of brightness orsaturation. Of course, various known processing may be adopted inaddition to these processing.

In any case, the function f for calculating the value indicating thedegree of fading is defined in advance, and the outputs F_(i1) to F_(iN)thereof are the input values for the layer L_(F). The number of values(the number of functions f) indicating the degrees of fading may be oneor more, and the number of nodes of the layer L_(F) is 1 when the degreeof fading is expressed by one kind of value for one photograph. In anycase of FIGS. 3 and 4, the range of input data may not be the entirerange of the photograph, and may be a specific color channel. A weightmay be provided for the specific color channel.

In the present specification, hereinafter, an embodiment will bedescribed by using the model in which the degree of fading is calculatedby the function f based on the photographic data and the valueindicating the degree of fading is input to the node of the neuralnetwork as an example illustrated in FIG. 4. It will be assumed in thisexample that the model in which the imaging time of the photograph isexpressed by a predetermined period of time (1900s or the like). Ofcourse, the imaging time of the photograph may be expressed in variousformats. In the example illustrated in FIG. 4, when informationindicating the imaging time is a period of time of a predetermined rangeand when the output values (T_(i1) to T_(i0)) of the output layer Lo isclosest to 1, it is estimated that the period of time associated withthis node is the imaging time. For example, when 1900s is associatedwith the node that outputs the output value T_(i1) and a period of timeindicated by each node is expressed every 10 years, an output value ofthe node associated with 1900s is 0.95, and the imaging time isestimated as 1900s when an output value of another node is a valuesmaller than 0.95 (the sum of the outputs of all the nodes is 1).

In step S100 in the flowchart illustrated in FIG. 2, the training modelis obtained. In this example, the training refers to a learning target.That is, the degree of fading is calculated from the photographic dataand the imaging time of the photograph is output based on the degree offading in the training model. However, a correspondence between thephotographic data and the imaging time of the photograph is not accurateat the beginning. That is, the number of layers constituted by the nodesor the number of nodes is determined in the training model, but theparameters (the weights, the biases, or the like) for defining therelationship between the inputs and the outputs are not optimized. Theseparameters are optimized (that is, trained) during the machine learning.

The training model may be determined in advance or may be obtained bybeing input by the user who operates the mouse 3 a or the keyboard 3 b.In any case, the processor 10 obtains the function for calculating thedegree of fading from the photographic data in the example illustratedin FIG. 4 and the parameters of the neural network that outputs theimaging time of the photograph by inputting the degree of fading as thetraining model. The training model is stored as a training model 20 e inthe HDD 20.

Subsequently, the processor 10 obtains training data items by thefunction of the learning target obtaining unit 11 a (step S105). In thepresent embodiment, training data items 20 c are training data itemsobtained by associating the photographic data items of the scannedphotographs with the imaging times of the photographs. In the presentembodiment, a plurality of photographs is scanned by the scanner 2 inadvance, and is stored in the HDD 20. The processor 10 can perform afunction of reading the imaging dates imprinted on the photographs (areading unit 12 c to be described below).

In this example, the processor 10 detects a portion having a shapefeature of numerical values based on the photographic data, andspecifies the arrangement of numerical values or based on the shapefeature of the detected portion. The arrangement of the numerical valuesis deemed to be the imaging date, and the imaging time is obtained. Whenthe imaging time is obtained, the processor 10 associates the imagingtime with the photographic data of the photograph. When the userremembers the imaging time of the photograph, the user specifies theimaging time of the scanned photograph based on their remembrance, andinputs the imaging time by operating the mouse 3 a or the keyboard 3 b.The processor 10 associates the photographic data with the imaging timebased on the input result.

As a result, the photographic data items of the plurality of photographsassociated with the imaging times are stored in the HDD 20. Theinformation indicating the imaging time may be a time including theimaging date of the photograph, may be defined in various formats, andmay match with at least the format of the imaging time expressed by theoutput of the training model. In the example illustrated in FIG. 4, theimaging time expressed by a predetermined period of time such as 1900sor the like is associated with the photographic data. In the presentembodiment, some of the photographic data items associated with theimaging times in this manner become the training data items. Thus, theprocessor 10 randomly extracts the photographic data items associatedwith the imaging times while referring to the HDD 20, deems theextracted photographic data items to be the training data items, andstores the extracted photographic data items in the HDD 20 (trainingdata items 20 c).

Subsequently, the processor 10 obtains test data by the function of thelearning target obtaining unit 11 a (step S110). In the presentembodiment, some or all of the data items which are not deemed to be thetraining data items, among the photographic data items associated withthe imaging times, become the test data items. Thus, the processor 10obtains the test data items from the data items which are not deemed tobe the training data items, among the photographic data items associatedwith the imaging times, while referring to the HDD 20, and stores theobtained test data items in the HDD 20 (test data items 20 d). Theamount of information (the number of photographs) of the test data items20 d and the amount of information (the number of photographs) of thetraining data items 20 c may be various amounts. In the presentembodiment, the amount of information of the training data items 20 c isset to be larger than the amount of information of the test data items20 d.

Subsequently, the processor 10 determines an initial value by thefunction of the learning unit 11 b (step S115). That is, the processor10 gives the initial value to a variable parameter among the trainingmodel obtained in step S100. The initial value may be determined byvarious methods. For example, a random value or zero can be used as theinitial value, and the initial value may be determined based on an ideathat the initial value is different for the weight and the bias. Ofcourse, the initial value may be adjusted such that the parameter isoptimized during learning.

Subsequently, the processor 10 performs the learning by the function ofthe learning unit lib (step S120). That is, the processor 10 inputs thetraining data items 20 c obtained in step S105 to the training modelobtained in step S100, and calculates the outputs indicating the imagingtimes. An error is specified by a loss function indicating errorsbetween the output imaging times and the imaging times indicated by thetraining data items 20 c. The processor 10 repeats processing forupdating the parameter based on differential using a parameter of theloss function by a predetermined number of times.

For example, in the model illustrated in FIG. 4, the processor 10obtains the degrees of fading (F_(i1) to F_(iN)) of an i-th photographby inputting the photographic data of the i-th photograph indicated bythe training data items 20 c obtained in step S105 to the function f.The processor 10 inputs the degrees of fading (F_(i1) to F_(iN)) of thei-th photograph to the neural network of the layer L_(F), and obtainsoutput values (T_(i1) to T_(i0)) indicating the imaging times of thei-th photograph. The information items indicating the imaging timesassociated with the training data items 20 c are information itemsindicating that any of the output values of each node is 1. When theseinformation items are expressed as (t_(i1) to t_(i0)), a value of any ofthe t_(si) to do is 1, and other values thereof are zero.

Thus, when the output values of the training model are expressed byT_(i) and the imaging times indicated by the training data items 20 care expressed by t_(i), the loss function for the i-th photograph can beexpressed by L(T_(i), t_(i)). Of course, various functions can beadopted as the loss function. For example, a cross entropy error or thelike can be adopted. Processing for calculating the loss functiondescribed above is performed for all the photographs indicated by thetraining data items 20 c, and the loss function in one learning isexpressed by the average or sum thereof. For example, when the lossfunction is expressed by the sum, the total loss function E is expressedby the following Expression 2.

$\begin{matrix}{E\; = \;{\sum\limits_{i}{{\; L_{i}}\left( {T_{i},\; t_{i}} \right)}}} & (2)\end{matrix}$

When the loss function E is obtained, the processor 10 updates theparameter by a predetermined optimization algorithm, for example, astochastic gradient descent method.

When the updating of the parameter is performed by a predeterminednumber of times, the processor 10 determines whether or not thegeneralization of the training model is completed (step S125). That is,the processor 10 inputs the test data items 20 d obtained in step S110to the training model, and obtains the outputs indicating the imagingtimes. The processor 10 obtains the number of photographs for which theoutput imaging times and the imaging times associated with the test dataitems match, and obtains estimation accuracy by dividing the number ofphotographs by the number of photographs indicated by the test dataitems 20 d. In the present embodiment, the processor 10 determineswhether or not the generalization thereof is completed when theestimation accuracy is equal to or greater than a threshold value.

The validity of a hyperparameter may be verified in addition to theevaluation of generalization performance. That is, a hyperparameterwhich is a variable amount other than the weight and the bias. Forexample, in a configuration in which the number of nodes or the like istuned, the processor 10 may verify the validity of the hyperparameterbased on verification data. Through the same processing as step S110,the verification data may be obtained by extracting the verificationdata in advance from the photographic data items of the plurality ofphotographs associated with the imaging times and securing the extractedverification data items as data items which are not used in thetraining.

When it is not determined that the generalization of the training modelis completed in step S125, the processor 10 repeats step S120. That is,the process further updates the weight and the bias. Meanwhile, when itis determined that the generalization of the training model is completedin step S125, the processor 10 stores a learned model (step S130). Thatis, the processor 10 stores the training model as a learned model 20 bin the HDD 20.

According to the aforementioned configuration, it is possible toestimate the imaging time of any photograph by inputting anyphotographic data to the learned model 20 b and outputting informationindicating the imaging time. Accordingly, even though the imaging dateis not imprinted in the photograph, it is possible to estimate theimaging time when the photograph is captured. In the present embodiment,since the learning is performed based on the training data itemsobtained by associating the photographic data items with the imagingtimes of the photographs, it is possible to estimate the imaging time ofthe photograph even though the relationship between the feature of thephotograph and the imaging time is not clearly defined by humanjudgment. The function f for calculating the degree of fading of thephotographic data is included in the model illustrated in FIG. 4.However, it is possible to estimate the imaging time of the photographeven though at least the relationship between the degree of fading andthe imaging time is not explicitly defined. Particularly, since thedegree of fading of the photograph is greatly influenced by the storageenvironment of the photograph, it is possible to more accuratelyestimate the imaging time in a case in which the relationship betweenthe feature of the photograph and the imaging time is learned anddefined based on the photograph stored by the user as compared to a casein which a manufacturer defines the relationship between the feature ofthe photograph and the imaging time.

The data items obtained by associating the photographic data items withthe imaging times are included in the training data items in the presentembodiment by reading the imaging date imprinted in the photograph andchecking the imaging time. Accordingly, the data items obtained byassociating the photographic data items with the imaging timesaccurately indicating the imaging dates of the photographs are includedin the training data items. Thus, according to the training data items,it is possible to increase a possibility that the learned model 20 b isto accurately output the imaging times.

(3) Image Processing

Next, image processing performed by the processor 10 will be described.FIG. 5 is a flowchart illustrating the image processing. The imageprocessing is performed by the user when the user wants to performprocessing (organizing, editing, or the like) related to thephotographic data items. That is, the user issues an instruction toperform the image processing by operating the mouse 3 a or the keyboard3 b while viewing the display 4. The instruction may be issued byvarious methods, and corresponds to an activation instruction of anapplication program for scanning an image in the present embodiment.

When the image processing is started, the processor 10 displays a userinterface screen by the function of the obtaining unit 12 a (step S200).That is, the processor 10 draws the user interface screen includingpredetermined display items while referring to data items of icons orthe like (not shown) stored in the HDD 20 by the function of theobtaining unit 12 a (stores image data or the like in a RAM). Theprocessor 10 displays the interface screen on the display 4 byoutputting the drawing contents to the display 4 through the displayinterface 40.

FIG. 6 is a diagram illustrating an example of the interface screen. Inthe example illustrated in FIG. 6, buttons Bsc, Bsh, Bup, Bed, and Bdeindicating functions to be performed are displayed on an upper part ofthe screen. A region Z₁ indicating a storage location of thephotographic data indicating the photograph and a region Z₂ fordisplaying a minified image of the photographic data stored in thestorage location are provided on a left side of the screen. In theregion Z₂, since a scroll bar is displayed when all the photographsstored in the same storage location cannot be displayed in a list, theuser can search for a desired photograph according to an instructionusing the scroll bar even though there are a large amount ofphotographs. A region Z₃ for displaying one photograph selected from thephotographs displayed in the region Z₁ and a region Z₄ for instructing akey when the photographs are sorted are provided on a right side of thescreen. The sort key may be various keys, and includes the imaging timesin the present embodiment. In addition, a file name or another conditionmay be selected as the sort key.

The button Bsc is a button for issuing an instruction to perform ascanning function, the button Bsh is a button for issuing an instructionto share the photographs (for example, for transmitting the photographsby an e-mail), and the button Bup is a button for issuing an instructionto upload the photographs to a predetermined server. Hardware used forcommunication is omitted in FIG. 1. The button Bed is a button forissuing an instruction to edit the photographs (for adjusting brightnessor the like), and the button Bde is a button for issuing an instructionto delete the photograph. When the instruction other than the scanningfunction is issued, the user selects the photograph displayed in theregion Z₂ by the mouse 3 a or the keyboard 3 b, and instructs the buttonBsh or the like to perform the function for the selected photograph. Thephotograph instructed by the user is displayed in the region Z₃ by theprocessor 10. In the example illustrated in FIG. 6, an outer frame ofthe photograph selected in the region Z₂ is represented by a thick solidline, and the selected photograph is displayed in the region Z₃.

When the user instructs the button Bsc by using the mouse 3 a or thekeyboard 3 b, the processor 10 controls the device interface 30 to startthe scanning of the photographs using the scanner 2. In the presentembodiment, the user can sort the photographs displayed on the interfacescreen illustrated in FIG. 6, and can display the photographs in thesorted order. In the region Z₄, a sort key can be displayed in apulldown format. When the user selects the sort key, the processor 10sorts the photographs by the selected sort key, and redisplays thephotographs in the region Z2 in the sorted order. Although it has beendescribed in the example illustrated in FIG. 6 that four photographs aredisplayed in the region Z₂, the number of photographs to be displayedmay be any number.

The processor 10 performs the processing corresponding to the operationof the user on the user interface screen in the order illustrated inFIG. 5. Thus, the processor 10 receives the operation of the user (stepS205). That is, the processor 10 receives output information of themouse 3 a or the keyboard 3 b through the device interface 30, anddetermines whether the operation performed by the user is theinstruction for any of the buttons Bsc, Bsh, Bup, Bed, and Bde or theselection of the sort key in the region Z₄.

When the operation performed by the user is the instruction for thebutton Bsc, the processor 10 determines that the scanning instruction isperformed in step S210. When it is determined that the scanninginstruction is performed in step S210, the processor 10 scans thephotograph and obtains the photographic data by the function of theobtaining unit 12 a (step S215). That is, the processor 10 outputs acontrol signal to the scanner 2 through the device interface 30, andscans the photograph set by the user for the scanner 2. As a result, thescanner 2 outputs the photographic data obtained as the reading result,and the processor 10 obtains the photographic data through the deviceinterface 30. In the present embodiment, the processor 10 stores thescanned photographic data in a default storage location (folder) of theHDD 20, and displays the photograph indicated by the scannedphotographic data in the region Z₂. When the plurality of photographs istargets to be scanned, the user repeats the set of the photograph forthe scanner 2, and the processor 10 scans the plurality of photographs.

Subsequently, the processor 10 reads the imaging date imprinted in thephotograph by the function of the reading unit 12 c (step S220). Thatis, the processor 10 detects a portion having the shape feature of thenumerical values based on the photographic data of the scannedphotograph in step S215, and determines the numerical values based onthe shape feature of the detected portion. The arrangement of thenumerical values is deemed to be the imaging date, and the imaging timeis obtained. For example, when a photograph Z₂₁ illustrated in FIG. 6 isused, since the imaging date is Apr. 1, 1961, the processor 10 obtains1960s as the imaging time. The processor 10 determines whether or not anattempt to read the imaging date succeeds as stated above (step S225),and the processor 10 associates the imaging time (imaging date) with thephotographic data of the photograph when the attempt to read the imagingdate succeeds (step S240).

Meanwhile, when it is not determined that the attempt to read theimaging date succeeds in step S225, the processor 10 performspre-processing on the photographic data by the function of thecalculating unit 12 b (step S230). That is, in the present embodiment,the processor 10 estimates the imaging time of the photograph based onthe learned model 20 b. The learned model 20 b is configured to outputthe imaging time of the input photographic data according to thepredetermined model as stated above. For example, the format forinputting the photographic data needs to have a format capable of beinginput the aforementioned function f in the example illustrated in FIG.4. Thus, the processor 10 performs pre-processing (for example,magnifying, minifying, standardizing, or the like) such that thephotographic data has the format capable of being input to the learnedmodel 20 b.

Subsequently, the processor 10 performs the estimation using the learnedmodel by the functions of the calculating unit 12 b and the organizingunit 12 d (step S235). That is, the processor 10 obtains the learnedmodel 20 b from the HDD 20. The processor 10 uses the photographic datapre-processed in step S230 as the input data for the learned model 20 b,and obtains the output result of the learned model 20 b as the imagingtime. In the example illustrated in FIG. 4, the learned model 20 bobtains the degree of fading of the photograph based on the inputphotographic data, inputs the degree of fading to the neural network,and outputs the imaging time of the photograph. Accordingly, theprocessor 10 performs the estimation of the degree of fading using thecalculating unit 12 b and the estimation of the imaging time (theestimation of the degree of oldness) using the organizing unit 12 d instep S235.

Subsequently, the processor 10 stores the photographic data inassociation with the imaging time by the function of the organizing unit12 d (step S240). That is, the processor 10 stores the photographic datain association with the information indicating the imaging timeestimated in step S235 in the HDD 20 (photographic data 20 a). Throughthe aforementioned processing, the photographic data 20 a obtained byassociating the imaging time with the scanned photograph is generated.Through the aforementioned processing, in the present embodiment, it ispossible to deem the method of generating the photographic data 20 aassociated with the imaging time to be performed by the computer 1. Theassociation of the information indicating the imaging time with thephotographic data may mean that the imaging time is written an internalregion (for example, a header) of a photograph file including thecorresponding photographic data or the imaging time is written in animaging time file different from the photographic data.

Thereafter, the processor 10 draws an image on which the photograph isadded in the order corresponding to the sort key selected by the sortkey in the region Z₄ to be described below by the function of theorganizing unit 12 d, and displays the drawn image on the display 4.

Meanwhile, when the operation performed by the user is the selection ofthe sort key in the region Z₄, the processor 10 determines that thesorting instruction is performed in step S245. When it is determinedthat the sorting instruction is performed in step S245, the processor 10sorts the photographs based on the sort key by the function of theorganizing unit 12 d (step S250). That is, the user selects the sort keyin the region Z4, and sorts the photographs based on the selected sortkey while referring to the information associated with the photographicdata 20 a. When the sorting is performed, the processor 10 draws theimage on which the photographs are lined up in the sorted order, anddisplays the drawn image on the display 4.

In the present embodiment, the user can select the imaging time as thesort key. When the imaging time is selected, the processor 10 selectsthe imaging time as the sort key, and sorts the photographs. That is, inthe present embodiment, the imaging time is associated with thephotographic data 20 a indicated by the scanned photograph by theprocessing in step S240. Thus, the processor 10 sorts the photographs indescending order or in ascending order (the descending order or theascending order may be selected by the user or may be determined inadvance) while referring the imaging times associated with thephotographic data items 20 a.

In the present embodiment, the imaging date is read in step S220, andthe read imaging time is associated with the read photographic data 20 ain step S240. Accordingly, not the imaging time estimated by the learnedmodel but the read imaging time is associated with the photograph. Thus,for the photograph of which the imaging date is read, the read imagingdate is selected as the sort key, and the imaging time (the imaging timeestimated based on the degree of fading) output by the learned model 20b is not selected as the sort key. Accordingly, when the sorting isperformed in the present embodiment, the read imaging date (the readimaging time) or the imaging date (the imaging time) input by the useris used in preference to the imaging time output by the learned model 20b.

When the imaging time is selected as the sort key in step S250 and thereis photographic data 20 a which is not associated with the imaging time,the processor attempts to estimate the imaging time. When the attempt toestimate the imaging time succeeds, the processor sorts the photographsin the estimated imaging time, and skips the sorting of the photographsusing the corresponding photographic data 20 a.

Meanwhile, when the operation performed by the user is anotherinstruction other than the sorting instruction (when the operationperformed by the user is any of the buttons Bsh, Bup, Bed, and Bde), theprocessor 10 performs another processing corresponding to theinstruction button (step S255). That is, the processor 10 performs theprocessing corresponding to the instruction according to the instructionof the user. For example, when the instruction using the button Bsh isperformed, the processor 10 displays an input screen of informationrequired for sharing the selected photograph on the display 4. When theuser issues an instruction to share the photograph after the informationis input, the photograph is shared with a person designated as a sharingtarget.

When the instruction using the button Bup is performed, the processor 10transmits the photographic data 20 a of the selected photograph to apredetermined server (not shown) through a communication unit (notshown). When the instruction using the button Bed is performed, theprocessor 10 displays an input screen of information required forediting the selected photograph on the display 4. When the user issuesan instruction to edit the photograph after the information is input,the photographic data 20 a is updated in a state in which the editedcontent is reflected. This editing includes editing such as trimming forthe image or editing of the information associated with the image. Whenthe user remembers the imaging time of the photograph, the user inputsthe imaging time through this editing. When the user inputs the imagingtime, the learning and the estimation may be reperformed according tothe confirmation of the input of the imaging time. When the instructionusing the button Bde is performed, the processor 10 deletes thephotographic data 20 a of the selected photograph from the HDD 20.

According to the present embodiment, it is possible to estimate animaging time of any photograph based on the learned model 20 b. Sincethe learned model 20 b is configured to output the imaging time of thephotograph based on the feature (the degree of fading or the like) ofthe photograph, it is possible to estimate the imaging time of thephotograph even though the imaging date is imprinted in the photograph.Accordingly, it is possible to organize the photographs according to thedegrees of oldness by organizing (sorting or the like) the photographsbased on the output imaging time. In the present embodiment, the imagingtime of the photograph is estimated based on the learned model 20 bduring sorting. Accordingly, the computer 1 functions as an imaging timeestimation apparatus that obtains the imaging time of the photograph inwhich the imaging date is not imprinted from the model on which machinelearning is performed based on the training data obtained by associatingthe photographic data of the scanned photograph and the imaging time ofthe photograph.

In the present embodiment, when the imaging date imprinted in thephotograph can be read, the sorting is performed while referring to theread imaging date (the read imaging time) in preference to the imagingtime estimated by the learned model 20 b. Accordingly, it is possible toorganize the photographs according to the degrees of oldness based onmore accurate information which is capable of specifying the imagingdates of the photographs more reliable than the degrees of fading.

(4) Other Embodiments

The aforementioned embodiment is an example for implementing the presentdisclosure, and other various embodiments can be adopted. For example,the machine learning apparatus and the image processing apparatusaccording to the embodiment of the present disclosure may be applied toan electronic device used for the purpose of use other than the reading,for example, a multi-function printer. The method of estimating theimaging time of the photograph in which the imaging date is notimprinted or the method of performing the learning for estimation basedon the photographic data as in the aforementioned disclosure may berealized as the disclosure related to a program or the disclosurerelated to a method.

The number of apparatuses constituting the image processing apparatusmay be any number. For example, the apparatus constituting the imageprocessing apparatus may be realized by an apparatus in which thecomputer 1 and the scanner 2 are integrated, or may be integrated withother various devices, for example, the display 4. The scanner 2 may becontrolled by a tablet terminal in which the computer 1 and the display4 are integrated.

A configuration in which the functions of the computer 1 are realized bya plurality of devices may be adopted. For example, a configuration inwhich a server and a client is connectable to each other, the machinelearning program is executed on one of the server and the client, andthe image processing program is executed in the other one thereof may beadopted. When the machine learning and the image processing areperformed in apparatuses present in separated positions from each other,the learned model may be shared between these apparatuses, or may bepresent in one apparatus thereof. When the learned model is present inthe apparatus that performs the machine learning and is not present inthe apparatus that performs the image processing, the apparatus thatperforms the image processing inquires of the apparatus that performsthe machine learning about the imaging time. Of course, a configurationin which the machine learning apparatus is present as a plurality ofdistributed apparatuses or a configuration in which the image processingapparatus is present as a plurality of distributed apparatuses may beadopted. The aforementioned embodiment is an example, and an embodimentin which a partial configuration is omitted or another configuration isadded may be adopted.

The obtaining unit may obtain the plurality of photographic data itemsacquired by scanning the photographs. That is, the obtaining unit mayobtain the photographic data items indicating the plurality ofphotographs as targets to be organized. Since the photographic data isalso a target for which the degree of fading is calculated, thephotographic data has a format in which the degree of fading can becalculated. This format may be various formats. For example, an examplein which the format is constructed by a plurality of color channels (R:red, G: green, B: blue, or the like) is used. Additional informationsuch as a generation date may be added to the photographic data. Sincethe photographic data is obtained through the scanning, a date when thephotographic data is generated is not limited to the imaging date unlikethe photographic data of the captured photograph. Accordingly, unlessthere are reasons that the imaging date is imprinted in the photograph,the imaging date of the photograph is known (for example, a case inwhich a note is written on a rear surface of the printed photograph orthe like), or the like, the imaging time of the photograph is unknown.The obtaining unit obtains at least one photograph which is notassociated with the imaging time.

The calculating unit may calculate the degree of fading of thephotograph for the photographic data. The degree of fading may be achange or a deterioration in color of the printed photograph. When thecolor of the printed photograph changes over time, the photograph fades.The degree of fading may be evaluated by various methods. In theaforementioned embodiment, as in a case in which the model in which thephotographic data is input and the imaging time is output isconstructed, even though it is unknown whether or not the degree offading is explicitly calculated, since the degree of fading isassociated with the degree of oldness, it is possible to deem the degreeof fading to be calculated within the model. That is, it is possible todeem the output including the calculation of the degree of fading to beperformed by the learned model.

The organizing unit may organize the photographic data items accordingto the degrees of oldness by deeming the photograph of which the degreeof fading is larger to be the old photograph. That is, in general, thelonger the elapsed time after the photograph is printed, the larger thedegree of fading. Thus, the organizing unit deems the photograph ofwhich the degree of fading is larger to be the old photograph.

The degree of oldness is the time interval between the date when thephotograph is printed and the current date. The longer the time intervalbetween the printed date and the current date, the older the photograph.Since the printed photograph starts to fade after the photograph isprinted, when the photograph is printed after a long time elapses fromthe imaging date, there may be a little relationship between the degreeof fading and the degree of oldness of the imaging date. The degree offading may change depending on a storage condition of the printedphotograph. Accordingly, there may be a photograph having a strongrelationship and a photograph having a weak relationship between thedegree of oldness of the photograph and the degree of oldness of theimaging date. However, in many cases, even though the imaging date andthe printed date are close to each other and the degree of oldnessevaluated with the degree of fading is the degree of oldness of theimaging date, when the photographs are organized according to thedegrees of oldness, an effect capable of sorting the photographs in asequence of time without spending time and labor inputting the imagingtime by the user is acquired.

The organizing in the organizing unit may be processing for givingvarious orders to the plurality of photographs based on the degrees ofoldness. Accordingly, the organizing may correspond to variousprocessing in addition to the sorting as in the aforementionedembodiment. For example, the organizing may be filtering using thedegree of oldness, may be adding the additional information to thephotographic data (giving the imaging date or the imaging time or thelike), and may be giving the file name to the photographic data,classifying the photographic data, storing the photographic data in thefolder, moving the photographic data, deleting the photographic data, orprocessing the photographic data. Various processing is used as theorganizing. Of course, the sorting or the filtering is not limited tothe displaying of the photographic data. Other processing, for example,adding the additional information or giving the file name may be used.

The reading unit may read the imaging date imprinted in the photograph.Of course, the reading unit may read a part of the imaging date. Forexample, when it is not possible to read the date of the numericalvalues indicating the imaging date due to their unclearness and it ispossible to read an imaging year and an imaging month, the reading unitmay read the imaging year and month. The imaging date may be imprintedin the photograph. That is, the imaging year, month, and date expressedby the numerical values of the color which is hard to be used for thephotograph may be printed near the side or corner of the printedphotograph.

The reading using the reading unit may be realized by varioustechnologies for recognizing the numerical values. Accordingly, asstated above, a technology for recognizing the imaging date by obtainingthe shape feature of the numerical values may be used, or a technologyfor recognizing the numerical values from the imaging through themachine learning. Various configurations for recognizing the numericalvalues can be adopted. When the numerical values are recognized, it ispreferable that the imaging date is selected based on the position ofthe numerical values due to the arrangement of the numerical valueswhich may be the date. The arrangement of the numerical values which maybe the date is an arrangement such as YYYY/MM/DD, and is not an unlikelydate such as the representation of February 35.

When the photographs are organized according to the imaging date inpreference to the degree of fading, the read imaging date may bereflected on the organizing in preference to the degree of oldnessestimated from the degree of fading. The organizing of the photographsaccording to the imaging date in preference to the degree of oldnessmeans that the same photograph is organized based on the imaging datewhen the imaging date and the degree of oldness estimated from thedegree of fading are obtained for the same photograph. For example, whenthe processing as the organizing is the sorting and the imaging date andthe degree of oldness estimated from the degree of fading are obtained,the photograph is sorted based on the imaging date.

The imaging time may be the capturing time, may be the imaging date, ormay be a longer period of time (for example, an imaging month, animaging year, or the like). Of course, the imaging time may be a timeclassified according to another method, for example, a time over aplurality of years such as 1980s or around 1980, a time specified by arange, may be a time interval (40 years ago or the like) from thecurrent date, or the like.

In the aforementioned embodiment, the organizing unit 12 d obtains thedegree of oldness of the photograph, that is, the imaging time of thephotograph based on the output of the learned model 20 b. However, thedegree of oldness of the photograph may be determined by another method.For example, a configuration in which the degree of oldness of thephotograph is determined based on a predetermined reference may beadopted, or the photograph may be organized by deeming the degree offading to match the degree of oldness.

Specifically, in the embodiment illustrated in FIG. 1, a configurationin which the processor 10 determines the degree of oldness of thephotograph by performing the image processing illustrated in FIG. 7 canbe adopted. The image processing illustrated in FIG. 7 is processing inwhich a part of the image processing illustrated in FIG. 5 is changed.The same processing in FIG. 7 as the processing in FIG. 5 will beassigned the same reference numeral. In this example, the description ofthe same processing as the processing in FIG. 5 will be omitted, andprocessing different from the processing in FIG. 5 will be mainlydescribed.

In this example, the processor 10 reads the imaging dates from thephotographs in which the imaging dates are imprinted by the function ofthe reading unit 12 c in step S220 (hereinafter, the photographs inwhich the imaging dates are imprinted are referred to as firstphotographs). Subsequently, the processor 10 determines whether or notthe attempt to read the imaging date succeeds (step S225). When theattempt to read the imaging date succeeds, the processor 10 stores thephotographic data in association with the imaging time by the functionof the organizing unit 12 d (step S226). That is, the processor 10stores the photographic data in association with the informationindicating the imaging time (the imaging date) estimated in step S235 inHDD 20 (photographic data 20 a).

When it is not determined that the attempt to read the imaging datesucceeds in step S225, the processor 10 skips step S226. In this case,the processor 10 deems the imaging dates not to be imprinted in thescanned photographs (hereinafter, the photographs in which the imagingdate is not imprinted is referred to as second photographs).

Subsequently, the processor 10 calculates the degree of fading of thephotograph by the function of the calculating unit 12 b (step S227).Various methods may be adopted as the calculation method of the degreeof fading. For example, a configuration in which the degree of fading iscalculated by the aforementioned function f, a configuration in whichthe degree of fading is calculated based on the model obtained by themachine learning based on the photographic data associated with thedegree of fading, or the like is used. When the degree of fading iscalculated by the function of the calculating unit 12 b, the processor10 stores the photographic data 20 a of each photograph in associationwith the information indicating the degree of fading in the HDD 20 (stepS228).

When the processor 10 generates the photographic data 20 a as statedabove, it is possible to estimate the imaging time of the photographbased on the photographic data 20 a. In the present embodiment, theprocessor 10 estimates the imaging time of the photograph, and sorts thephotographs in the order of the estimated imaging times when thephotographs are sorted in step S251. That is, in the present embodiment,the processor 10 compares the degrees of fading of the first photographsin which the imaging dates are imprinted with the degrees of fading ofthe second photographs in which the imaging dates are not imprinted,determines the degrees of oldness of the first photographs and thesecond photographs, and organizes the photographs.

Specifically, the processor 10 uses all the second photographs assequential determination targets, and specifies the first photographs ascomparison targets for each determination target. The processor 10compares the degrees of fading of the first photographs as thecomparison targets with the degrees of fading of the second photographsas the determination targets. When the degrees of fading of the secondphotographs are larger than the degrees of fading of the firstphotographs, the processor 10 determines that the second photographs asthe determination targets are older than the first photographs as thecomparison targets. When the degrees of fading of the second photographsare smaller than the degrees of fading of the first photographs, theprocessor 10 determines that the first photographs as the comparisontargets are older than the second photographs as the determinationtargets. According to the aforementioned configuration, it is possibleto determine the degree of oldness of the photograph of which theimaging date is unknown by using the photograph of which the imagingdate is known as a reference.

The comparison target to be compared with the second photograph may beselected by various methods. For example, the first photograph of whichthe degree of fading is closest may be the comparison target, theplurality of first photographs of which the degrees of fading are closemay be the comparison targets, or a specific first photograph may be thecomparison target. Other various methods may be adopted. In the presentembodiment, since the imaging times are specified in the firstphotographs, the processor 10 sorts the first photographs based on theimaging times. The processor 10 sorts the photographs by inserting thesecond photographs of which the degrees of oldness are specified byusing the first photographs as the reference in the order of thephotographs in a sequence of time by using the sorted first photographsas the reference.

According to the aforementioned configuration, it is possible to sortthe second photographs of which the imaging times are not confirmedthrough the reading by using the sorted first photographs of which theimaging times are confirmed through the reading as the reference. Whenthe sorting is performed, the processor 10 draws the image on which thephotographs are lined up in the sorted order, and displays the drawnimage on the display 4. When it is not possible to determine therelationship between the degrees of oldness for all the photographsthrough the comparison of the first photographs with the secondphotographs, the relationship between the degrees of oldness may bedetermined by comparing the degrees of fading of the second photographs.

The imaging time may be estimated in addition to the relationshipbetween the degrees of oldness based on the degrees of fading. Thisconfiguration can be realized by a configuration in which the processor10 estimates the imaging times of the second photographs based on thedegrees of fading of the first photographs and the degrees of fading ofthe second photographs by the function of the organizing unit 12 d whilereferring to the photographic data items 20 a. Specifically, aconfiguration in which the processor 10 associates the imaging times(the imaging dates) of the first photographs with the degrees of fadingof the first photographs and deems the second photographs of which thedegrees of fading are similar to have same imaging time can be adopted.

When the degrees of fading are different, the processor 10 may deem theimaging times to be different. When there are two or more firstphotographs, the imaging time corresponding to any degree of fading maybe specified based on a difference between the degrees of fading of thefirst photographs. For example, a configuration in which when a degreeof fading of a certain second photograph is between the degrees offading of two first photographs, the second photograph is deemed to becaptured between the imaging times of these first photographs can beadopted. Of course, the imaging times of the second photographs may beestimated from a difference between the imaging times corresponding to adifference between the degree of fading of the first photograph and thedegree of fading of the second photograph by deeming the differencebetween the degrees of fading of two first photographs to correspond tothe difference between the imaging times of the first photographs.According to the aforementioned configuration, it is possible toestimate the imaging time of the photograph of which the imaging date isunknown by using the photograph of which the imaging date is known asthe reference.

The learning target obtaining unit may obtain the plurality ofphotographic data items of the scanned photographs. That is, thelearning target obtaining unit may obtain the photographic data items ofthe plurality of photographs as machine learning targets. Thephotographic data may be the training data to be referred duringlearning, or the photographic data may be associated with the imagingtime in advance. Alternatively, the imaging time may be estimated basedon the photograph, and the photographic data may be associated with theestimated imaging time. The latter case may be performed by variousmethods. For example, a configuration in which the degree of fading ofthe photograph is specified for each of the photographs of which theimaging times are known and the photographs of which the imaging timesare unknown and the imaging times of the photographs of which theimaging times are unknown are estimated based on the degrees of fadingcan be adopted. Of course, the estimation method of the imaging time maybe performed by another method, or the imaging time may be designated bythe user based on the note of the rear surface of the photograph or theremembrance of the user.

Since the photographic data is the target for learning the imaging time,the photographic data is data which indirectly includes at least theinformation indicating the imaging time. For example, since theestimation in which the degree of oldness and the imaging time of thephotograph are deemed to substantially match is a reasonable estimation,it is preferable that the degree of oldness of the photograph, forexample, the degree of fading is reflected on the photographic data.Thus, the photographic data has a format capable of calculating thedegree of fading. This format may be various formats. For example, anexample in which the format is constructed by a plurality of colorchannels (R: red, G: green, B: blue, or the like) is used.

Of course, the pre-processing before the machine learning may beperformed on the photographic data. For example, when the size (thenumber of pixels or the like) of the photographic data to be inputduring learning is restricted, the size of the photographic data may beadjusted through the magnifying, the minifying, or the like such thatvarious photographic data items have a predetermined size. Normalizationfor converting a range of gradation values of the photographic data intoa predetermined range or the like may be performed, or various imageprocessing such as edge enhancement or gamma conversion may beperformed.

The learning unit may learn the imaging times of the photographs inwhich the imaging dates are not imprinted based on the photographicdata. That is, the learning unit may perform the machine learning basedon the photographic data and the imaging time of the photograph. Themethod of the machine learning may be various methods. That is, a modelin which the photographic data items and the values (the degrees offading or the like) derived from the photographic data items are inputand the imaging times of the photographs are output may be constructed,and leaning for minimizing a difference between the output from themodel and the imaging time of the training data may be performed.

Accordingly, when the machine learning using the neural network isperformed, the machine learning may be performed by appropriatelyselecting various elements such as the number of layers constituting themodel, the number of nodes, the kind of the activation function, thekind of the loss function, the kind of the gradient descent method, thekind of the optimization algorithm of the gradient descent method,whether or not to perform mini-batch learning or the number of batches,the learning rate, the initial value, the kind of an over learningsuppression method, or whether or not to perform the over learningsuppression method, the presence or absence of a convolution layer, thesize of a filter in a convolution operation, the kind of the filter, thekind of padding and stride, the kind of a pooling layer, the presence orabsence of the pooling layer, the presence or absence of a total bindinglayer, and the presence or absence of a recursive structure. Of course,another machine learning, for example, learning such as support vectormachine, clustering, or reinforcement learning may be performed.

The machine learning in which the structure (for example, the number oflayers, the number of nodes for each layer, or the like) of the model isautomatically optimized may be performed. The learning may be dividedinto a plurality of stages and may be performed. For example, aconfiguration in which the machine learning for learning the degree offading from the photographic data and the machine learning for learningthe imaging time from the degree of fading are performed may be adopted.In a configuration in which the machine learning is performed by theserver, the training data items may be collected from a plurality ofclients and the machine learning may be performed based on thesetraining data items.

The photograph in which the imaging date is not imprinted may be in astate in which the numerical values deteriorate due to the fading andcannot be read, in addition to a state in which the numerical valuesindicating the date are not printed near the side or corner of theprinted photograph. That is, a state in which the numerical valuescannot be read is equivalent to a state in which the imaging date is notimprinted. Since the imaging date of the photograph in which the imagingdate is imprinted can be accurately specified by reading the imagingdate, it is not necessary to estimate the imaging date of thisphotograph. However, this photograph may be added to a sample forlearning.

The imaging time may be a time estimated as the time when the photographis captured, may be an imaging date, may be a longer period of time (forexample, an imaging month, an imaging year, or the like), may be a timeover a plurality of years such as 1980s or around 1980s, may be a timespecified by a range, or may be a time interval (40 years ago or thelike) from the current date. Of course, the imaging time may bespecified by a continuous value (for example, probability, reliability,or the like), or may be specified by a discrete value (for example, agraph indicating any of a plurality of candidates or the like).

When the learning is performed based on the training data obtained byassociating the photographic data with the imaging time of thephotograph, the photographic data and the imaging time may be the inputand the output for the model such as the neural network, or the values(for example, the values indicating the degrees of fading or the like)obtained by processing the inputs may be the input and the output.

The learning or the estimation of the imaging time may be performedbased on a clarification instruction from the user. For example, abutton of “estimation of imaging time” may be displayed, and thelearning and the estimation may be performed by operating this button.

The estimation of the imaging time is not limited to the estimationbased on the degree of fading, and may be performed by using anotherelement. For example, the estimation may be performed based on thedegree of fading and a degree of deterioration of photograph paper.

The present disclosure is also applicable to a program or a methodexecuted by the computer. The aforementioned program or method may berealized as a single apparatus, or may be realized by using componentsof a plurality of apparatuses. The aforementioned program or method mayinclude various aspects. The aforementioned program or method may beappropriately changed like a case where a part thereof is software orhardware. The present disclosure is established as a storage medium of aprogram. Of course, the storage medium of the program may be a magneticstorage medium, or may be a semiconductor memory or the like.Alternatively, any storage medium to be developed in the future can besimilarly considered.

What is claimed is:
 1. A machine learning apparatus comprising: aprocessor that obtains a plurality of photographic data items ofphotographs, with the photographic data items of the photographs beingacquired by scanning the photographs, the processor training a machinelearning model that associates a photographic data item of a photographor a data item derived from the photographic data item with a shootingtime of the photograph based on the photographic data items of thephotographs that have been obtained.
 2. The machine learning apparatusaccording to claim 1, wherein the processor trains the machine leaningmodel based on training data items that associate the photographic dataitems of the photographs that have been obtained with shooting times ofthe photographs.
 3. The machine learning apparatus according to claim 2,wherein the training data items include a training data item thatassociates a photographic data item acquired by scanning a photograph onwhich a date stamp is imprinted with a shooting time indicated by thedate stamp.
 4. An imaging time estimation apparatus comprising: aprocessor that obtains a shooting time of a photograph on which a datestamp is not imprinted from a machine leaning model on which machinelearning is performed based on training data items that associatephotographic data items of photographs with shooting times of thephotographs, with the photographic data items of the photographs beingacquired by scanning the photographs.
 5. A generation method comprising:obtaining a plurality of photographic data items of photographs, withthe photographic data items of the photographs being acquired byscanning the photographs; generating a machine learning model thatassociates a photographic data item of a photograph or a data itemderived from the photographic data item with a shooting time of thephotograph by performing machine learning based on the photographic dataitems of the photographs that have been obtained; and outputting ashooting time of a photograph on which a date stamp is not imprintedbased on the machine learning model.